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BACKGROUND OF THE INVENTION 

Field of the Invention 

[0001] The present invention relates generally to networked printing systems and, 
more specifically, to document printers that can receive, process, and transform 
multimedia data from a peripheral device. 

Description of the Related Art 

[0002] A conventional printer can receive documents or other data in a number of 
formats and then print the contents of those documents or data in accordance with the 
proper format. But while conventional printers can print documents in a wide variety of 
formats, these printers are fundamentally limited in their ability to take in contents from 
multimedia devices such as video cameras and cellular phones and process the data to 
create a useable record. For example, it is standard technology for a printer to produce 
images of static text, pictures, or a combination of the two. 

[0003] Meanwhile, cost and quality improvements in multimedia technologies have 

led to a proliferation of digital devices with multimedia capabilities. High-quality video 

cameras and cellular phones with multimedia capabilities are commonplace in the home 
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and workplace, and are useful for diverse purposes ranging from teleconferencing to 
managing information. Multimedia data captured by such devices are typically 
delivered in an unprocessed form to a storage medium such as a digital tape or memory 
card. 

[0004] Thus, there is a need for an integrated printer that can receive multimedia 
data from a peripheral device, process it, and deliver an output to a printed document or 
other media. It is further desirable that such a printer be able to perform at least some of 
the necessary processing itself , while some of the processing may be performed on an 

r 

external device, rather than requiring an attached computer or other device to perform 
all of the processing. 

SUMMARY OF THE INVENTION 

[0005] The present invention overcomes the deficiencies and limitations of the prior 
art by providing networked systems and methods in which multimedia data from 
peripheral devices are received by a printer, the data are processed, and the printer 
outputs the result. 

[0006] In a typical hardware configuration for such a multifunction printer, a printer 
includes a peripheral interface that communicates with a peripheral device, a print 
engine that produces a paper or other printed output, and one or more electronic 
devices that produce a related electronic output. Together, the printed and electronic 
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outputs provide an improved representation of the multimedia data from the peripheral 
device compared to a convention paper printer. 

[0007] In one embodiment, a networked printing system has a network interface 
coupled to a network device via the network. The printing system also includes a 
peripheral interface coupled to a peripheral device. The interfaces receive multimedia 
data from a peripheral device (either directly coupled or via the network), and a 
multimedia processing system is coupled to the interfaces for processing the multimedia 
data. Based on any of a number of desired applications, the multimedia processing 
system determines a printed representation of the multimedia data and an electronic 
representation of the multimedia data. To share the computing load, the multimedia 
processing system resides at least in part on the printing system and at least in part on 
the network device. A printed output system in communication with the multimedia 
processing system receives the printed representation and produces a corresponding 
printed output. Similarly, an electronic output system in communication with the media 
processing system receives the electronic representation and produces a corresponding 
electronic output. In this way, the printer creates a representation of multimedia data 
from a peripheral device by producing a printed output and an electronic output. 
[0008] In various embodiments, the system includes hardware and software for 
processing multimedia contents and various mechanisms for creating the electronic and 
printed outputs. For example, the interfaces may include a single communication 
interface, a network interface, a wireless interface, personal digital assistant (PDA) 
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device, a cellular phone, a removable media storage device reader, a video input device 
(such as a DVD reader or a video cassette reader), an audio input device (such as a CD 
reader or an MP3 player), a digital video recorder (e.g., TiVO), screen capture hardware, 
a video and/or audio recorder, or any of a number of different types of devices that can 
receive multimedia data. Similarly, the electronic output system may write the 
electronic representation to one or more different types of removable media storage 
devices, such as a DVD, a digital video recorder, a digital audio recorder, video cassette 
tape, a CD, an audio cassette tape, a flash card, a computer disk, an SD disk, or another 
computer-readable medium. The electronic output system may also include a 
disposable media writer, a self-destructing media writer, a video display, an audio 
speaker, a driver for a speaker system (such as an embedded MIDI player), or an 
embedded web page display. In this way, a multifunction printer can be configured to 
process any of a large number of multimedia data from various peripheral devices, 
allowing various embodiments of the printer to meet the needs of many different 
applications. 

[0009] Because of the great many combinations of input and output devices possible 
for the networked printing system, the system may include embedded or peripheral 
hardware, software, or a combination thereof for performing a wide variety of different 
operations on the multimedia data. In this way, the system can be configured to 
produce various, types of printed and electronic outputs based on received multimedia 
data to meet the needs of different applications. To solve various problems, in 
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embodiments of the system, the multimedia processing system includes one or more of 
an embedded multimedia module. 

[0010] Moreover, the processing logic in the printing system can be configured to 
communicate with a peripheral device via the peripheral interface and/or the network 
interface. Thus, the processing logic has the capability to operate the peripheral device 
and transfer data to the peripheral device. 

[0011] These different tasks may be performed on the printer by the multimedia 
processing system, or partially on the printer by the multimedia processing system in 
conjunction with one or more electronic devices capable of performing some of the 
required processing steps. The printer can thus balance the required processing of the 
multimedia data between the printer and one or more connected electronic devices, such 
as a personal computer or an external network service. By conducting at least some of 
the processing on the printer, the printer relieves at least some of the processing load 
from external devices that the printer's additional functionality may require. 
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Brief Description of the Drawings 
[0012] FIG. 1 is a schematic diagram of a system in accordance with an embodiment 
of the invention. 

[0013] FIG. 2 is a flow diagram of the operation of a system in accordance with an 
embodiment of the invention. 

[0014] FIG. 3 is a schematic diagram of various peripheral interfaces of a printing 
system, in accordance with embodiments of the invention. 

[0015] FIG. 4 is a schematic diagram of the printer's printed output system, in 
accordance with an embodiment of the invention. 

[0016] FIG. 5 is a schematic diagram of various electronic media output systems of 
the printer, in accordance with embodiments of the invention. 
[0017] FIG. 6 is a schematic diagram of a multimedia module of a printer, in 
accordance with an embodiment of the invention. 

[0018] FIG. 7 is a flow diagram of the operation of a system for automatically 
receiving multimedia data from a docked peripheral device, in accordance with an 
embodiment of the invention. 

[0019] FIG. 8 is an illustration of interactive communication with a printer in 
accordance with an embodiment of the invention. 
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Detailed Description of the Preferred Embodiments 
[0020] Various embodiments of a networked printing system enable the printing of 
multimedia data in a useful and intelligent format To create a representation of this 
multimedia data, the printing system produces a printed output and a related electronic 
output, which together provide a representation of the received multimedia data. 
Depending on the desired application, the printing system includes a multimedia 
module with functionality for processing the multimedia data, printing the printed 
output, and producing the electronic output. Therefore, a number of embodiments of 
the printing system are described herein to show how such a system can be configured 
in a wide variety of combinations. 

System Architecture 

[0021] FIG. 1 is a diagram of one embodiment of a networked printing system, 
which includes a printer 100 in communication with a peripheral device 150 and a 
network device 170. A network 155 enables communication between the printing 
system 100 and the network device 170. Printing system 100 includes a peripheral 
interface 105, user interface 110a, a printed output system 115, an electronic output 
system 120, and a multimedia processing system 125. Capable of receiving multimedia 
data from a peripheral device 150 and a network device 170, the peripheral interface 105 
and network interface 157, respectively, can each take a variety of forms and may 
include one or more devices that can receive multimedia data or create multimedia data 
by observing a media event. Similarly, the printed output system 115 and the electronic 
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output system 120 can take a variety of forms and may each include one or more devices 
that can produce, respectively, a printed output 160 and an electronic output 170. 
[0022] In one embodiment, the multimedia processing system 125 includes a 
memory 130, a processor 135, and a multimedia processing module 140. The 
multimedia processing module 140, which is described in more detail below, may 
include software, hardware, or a combination thereof for implementing at least a portion 
of the functionality of the printing system 100. The multimedia processing system 125 is 
coupled to the peripheral interface 105 and network interface 157, allowing it to 
communicate with each. The media processing system 125 is also coupled to the printed 
output system 115 and to the electronic output system 120 for providing the appropriate 
commands and data to those systems. 

[0023] The printing system 100 further includes a network interface 157, 
functionally coupled to the multimedia processing system 125. The network interface 
157 allows the printing system 100 to communicate with other electronic devices, such as 
network device 170 and external service 180. In one embodiment, the network device 
170 is a computer system, such as a personal computer. Network device 170 includes 
processing capability for performing processing on the multimedia data. In this way, 
the network device 170 can relieve the printing system 100 of some of the processing 
load required to produce printed and electronic outputs from the multimedia data. In 
one embodiment, the network device 170 includes a user interface 110b that allows a 
user to make selections about the processing of the multimedia and/or about the format 
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or destination of the printed or electronic outputs. In other embodiments, the user 
interface 110b can be located on another attached device, or the user interface 110a can 
be located on printer 100 itself. The user interface 110 may include a display system, 
software for communicating with an attached display, or any number of embodiments 
described in co-pending U.S. patent application entitled, "User Interface for Networked 
Printer," filed March 30, 2004, Attorney Docket No. 20412-08456, which application is 
incorporated by reference in its entirety. 

[0024] In another embodiment, the printing system 100 is coupled to an external 
service 180, which includes hardware and/or software for performing some of the 
processing tasks on the multimedia to be printed. In a typical embodiment, a remote 
service provider operates the external service 180. In such an embodiment, whereas the 
network device 170 may communicate with the printing system 100 over a local area 
network, the external service may communicate with the printing system 100 over a 
wide area network or over the Internet. By sharing the media processing tasks with an 
external service 180, possibly operated by a service provider, the printing system can 
perform tasks that are under the control of the service provider. In this way, a service 
can be set up around a particular form of multimedia processing, where a user pays for 
use of the service. 

[0025] FIG. 2 shows an overview of a generalized process in which the printing 
system 100 creates a representation of multimedia data, in accordance with one 
embodiment of the invention. The system 100 first receives 205 multimedia data from a 
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peripheral device 150 via the peripheral interface 105, or alternatively from a network 
device 170 via the network interface 157. This multimedia data may be received as 
digital or analog data, or it may be an observable event that the interface records as 
digital or analog data. Coupled to the interfaces 105, 157 to receive the multimedia data, 
the multimedia processing system 125 processes 210 the data to generate printed and 
electronic outputs. This processing 210 is performed in accordance with the intended 
functionality of the printing system 100, and examples of different operations are 
described in greater detail below. 

[0026] In one embodiment, the system automatically generates a printed output or 
an electronic output of the multimedia data based on a predefined format and the 
functionality of the peripheral device 150 that is connected to the peripheral interface 
105 or network interface 157. For example, the multimedia processing system can take a 
video clip from a digital video camera connected to peripheral interface 105 and 
automatically generate a printed output 160 containing keyframes and bar codes for 
accessing video segments of the video file. 

[0027] In another embodiment, the system 100 includes a user interface 110 to allow 
a user to preview the generated outputs. If 215 the user desires to refine the processing, 
the user can enter commands, which the printing system 100 receives 220 by way of the 
user interface 110. Based on the user's commands, the printing system 100 then repeats 
the processing 210 of the media data to generate new printed and electronic outputs. 
This refinement process can be iterated until the user is satisfied with the printed and 



11 



electronic outputs. When this occurs, the user indicates that the printing should 
commence, for example, by invoking a print command with the user interface 110. The 
multimedia processing system 125 then sends the generated printed output to the 
printed output system 115 and the electronic output to the electronic output system 120. 
The printed output system 115 then creates 225 a printed output 160, and the electronic 
output system 120 creates 230 an electronic output 170. 

[0028] It can be appreciated that this generalized description of a multifunction 
printer lends itself to a great number of specific configurations and applications. 
Accordingly, examples of the possible configurations, applications, and particular 
components are further described. 

Peripheral Interface 

[0029] The peripheral interface 105 can be designed to accommodate any suitable 
type of multimedia peripheral device 150.. Because of the great variety of types and 
formats of multimedia data, the peripheral interface 105 may take any number of forms 
to accept any devices 150 that a user might wish to connect to print the multimedia data. 
FIG. 3 illustrates some examples of different interfaces 105 by which the printer 100 can 
receive multimedia data from a peripheral device or network device. In particular 
implementations, system 100 may have only one or only a subset of these types of 
interfaces 105, and in addition may have other types of interfaces not shown. 
[0030] As shown in FIG. 3, the printer 100 may include a communication interface 
305 that allows the printer 100 to be communicatively coupled to another electronic 
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device. Depending on the desired input, the interface 305 may allow the computer to 
communicate with a wide variety of different peripheral devices 150 that can provide 
the printer 100 multimedia data to print. Without intending to limit the types of devices, 
the interface 305 may allow the printer 100 to received media data from peripheral 
devices 150 such as computer systems, computer networks, digital cameras, cellular 
telephones, PDA devices, video cameras, media Tenderers (such as DVD and CD 
players), media receivers (such as televisions, satellite receivers, set-top boxes, radios, 
and the like), digital video recorders (such as a TiVO), a portable meeting recorder, 
external storage devices, video game systems, or any combination thereof. The 
connection type for the interface 305 can take a variety of forms based on the type of 
device that is intended to be connected to the printer 100 and the available standard 
connections for that type of device. For example, the interface 305 may comprise a port 
for connecting the device using a connection type such as USB, serial, FireWire, SCSI, 
IDE, RJ11, parallel port (e.g., bi-directional, Enhanced Parallel Port (EPP), Extended 
Capability Port (ECP), IEEE 1284 Standard parallel port), optical, composite video, 
component video, or S- video, or any other suitable connection type. 
[0031] In another embodiment, the printing system 100 includes a wireless interface 
310. As illustrated, the wireless interface 310 allows system 100 to receive multimedia 
data from a wireless device external to system 100. The wireless interface 310 may allow 
the system 100 to communicate with any number of wireless communication systems, 
such as wireless components on a home or business network, cellular phones and other 
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portable wireless devices, satellites, satellite dishes, and devices using radio 
transmissions. Depending on the types of external devices with which system 100 is 
communicating, the wireless interface 310 may comprise hardware and/or software that 
implements a wireless communications protocol, such as that described in IEEE 802.11, 
IEEE 802.15, IEEE 802.16, or the Bluetooth standard. 

[0032] In another embodiment, system 100 receives media data from a removable 
media storage reader 315 that is built into the printer 100. The removable media storage 
reader 315 may be configured to accommodate any type of removable media storage 
device, such as DVDs, CDs, video cassette tapes, audio cassette tapes, floppy disks, ZIP 
disks, flash cards, micro-drives, memory sticks, SD disks, or any other suitable type of 
multimedia storage device. Moreover, the printing system 100 may have a plurality of 
removable multimedia storage readers 315 to accommodate multiple types of media 
storage devices. 

[0033] In another embodiment, system 100 includes a docking station 335 that is 
built into the printer 100. The docking station 335 may be configured to accommodate 
any type of peripheral device, such as cell phones, digital audio recorders, video 
camcorders, portable meeting recorders, fixed position meeting recorders, head- 
mounted video cameras, office-based PC experience capture systems, or any other 
suitable type of multimedia peripheral devices. Moreover, the printer 100 may have a 
plurality of docking stations 335 to accommodate multiple types of peripheral devices. 
Furthermore, it will be understood that a peripheral device 150 may also be 
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communicatively coupled to the communication interface 305 via an external docking 
station. 

[0034] In another embodiment, the printer may include video capture hardware 355. 
In one embodiment, the video capture hardware 355 is designed to be coupled to a 
computing system by a video cable thereof. The video cable from a display is attached 
to the printer 100, where the video signal is split with one signal directed to the 
computing system and another signal to the video capture hardware 355. The video 
capture hardware 355 performs a differencing between successive frames of the video 
signal and saves frames with a difference that exceeds a threshold on a secondary 
storage in the printer 100. This offloads such processing from the computing system, 
thereby improving responsiveness and user experience and providing an easily 
browseable record of a user's activities during the day. To take advantage of the 
printing capabilities of the multifunction printer, the user can choose to print selected 
frames captured by the video capture hardware 355. The printing can be generated on 
demand with the user interface 110 on the printer or from the attached computing 
system, or automatically with scheduling software. In this way, a user can view a replay 
of any actions taken on the computing system. Notably, the captured content can be 
effectively compressed because the differences between frames are small. 
[0035] In another embodiment, the video capture hardware 355 is coupled to a 
converter module 360, such as VGA-to-NTSC conversion hardware. Such an 
embodiment could be used in conjunction with a projector to capture presentations 
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made with the projector. Audio capture could also be employed to record a speaker's 
oral presentation. To use the video capture hardware 355 in this way, a user could 
connect a laptop or other computing system and the projector to the printer 100. The 
printer 100 then captures video frames and compares them to the most recently captured 
frame and retains those frames that are different. A parallel audio track may also be 
saved. This capability could also be used in a desktop printer to record a presentation 
made on a computing system connected to the printer. The printer can then serve the 
audio itself or it can be written to a digital medium, such as an SD disk that can be 
played from a cell phone or a PDA. The audio could also be written to a bar code on a 
printed representation. 

Printed Output System 

[0036] The printed output system 115 may comprise any standard printing 
hardware, including that found in standard laser printers, inkjet printers, thermal wax 
transfer printers, dye sublimation printers, dot matrix printers, plotters, or any other 
type of printing mechanisms suitable for creating a printer image on an appropriate 
physical medium. In the example described herein, a laser printer mechanism is 
described; however, it should be understood that any suitable printing system can be 
used. The printing system 100 includes any necessary subsystems, as know by one 
skilled in the art, to print on a printable medium, such as a sheet of paper. 
[0037] In one embodiment, the printed output system 115 comprises a media supply 
handler 405 (FIG. 4) that receives blank paper to be printed on. The media supply 
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handler 405 typically obtains the paper from a supply tray 410. The printer 100 may 
include multiple supply trays 410, allowing the printer to accommodate different sizes 
and types of paper as well as trays 410 of varying capacity. When the printer 100 needs 
blank paper for printing, the media supply handler 405 provides the print engine 420 
with a sheet of blank medium. 

[0038] The formatter 415 converts data received from the multimedia processing 
system 125 into a format that the print engine 420 can use to create an image on the 
paper. The print engine 420 creates an image on the paper as indicated by the formatter 
415. A fuser 425 then uses high temperature and pressure to fuse the image onto the 
paper to fix the image thereon. Once the image is fixed, the paper is fed to the media 
output handler 430. Although not shown, it is appreciated that the printer 100 includes 
any necessary motors, gears, and diverters to cause the paper to move through the 
printer 100. 

[0039] The media output handler 430 receives one or more printed sheets of paper 
and performs any requested finishing to the sheets. For example, the media output 
handler 430 may include a sorter 435 to sort or collate the sheets for multiple copies and 
a stapler 440 to attach the sheets together. When the finishing process is complete, the 
media output handler 430 moves the sheets to an output tray 445, of which there may be 
multiple trays 445 to accommodate different sizes, types, and capacities of printed 
output. 

Electronic Output System 
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[0040] The electronic output system 120 can be designed to produce an electronic 
output related to the multimedia data in any desired format. Because of the great 
variety of types and formats of electronic outputs, the electronic output system 120 may 
take any of a number of forms for producing an electronic output desired by the user. 
FIG. 5 illustrates some examples of different embodiments of the electronic output 
system 120. In particular implementations, system 100 may have only one or only a 
subset of the various components shown, and in addition it may have other types of not 
shown. 

[0041] In one embodiment, the printer 100 writes the electronic output to a 
removable media device with a media writer 505. Many different types of media writers 
are known in the art, and the media writer 505 many comprise any of these. For 
example, the media writer 505 may be configured to write the electronic output to 
removable storage devices such as a writeable DVD or CD, a video cassette tape, an 
audio cassette tape, a flash card, a computer disk, an SD disk, a memory stick, or any 
other appropriate electronically-readable medium. Moreover, the electronic output 
system 120 may include a number of media writers 505 of different types to allow the 
printer 100 to print onto different electronic formats. In addition, the electronic output 
system 120 may include a number of media writers 505 of the same type to increase the 
output capacity of the printer 100. 

[0042] The removable storage device that receives the electronic output from the 
printer 100 may be fed to the media writer directly by a user, for example by inserting a 
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blank disk into a drive. In another embodiment, the printer 100 includes an electronic 
media handling mechanism 510 coupled to the media writer 505 that automatically 
provides the media writer 505 with an appropriate type of removable storage device. 
The handling mechanism 510 may further be configured to physically place written-to 
storage devices into an output tray 515. In one embodiment, a series of blank storage 
devices are fed to the printing system 100 by a bandolier 520 or other type of feeder, 
allowing the printer 100 to create a high volume of electronic output without requiring a 
significant amount of interaction with a human operator. The bandolier 520 preferably 
then places the written-to devices into an output tray 515. 

[0043] In another embodiment, the media writer 505 is a disposable media writer, 
configured to write electronic data to a disposable removable media storage mechanism. 
In another embodiment, the media writer 505 writes the electronic data to a self- 
destructing medium. In this way, a user can view the electronic data for a 
predetermined number of times or during a predetermined period of time, after which 
the electronic data are no longer viewable. 

[0044] In another embodiment, the electronic output system 120 includes a speaker 
system 530. The speaker system 530 is designed to receive an audio signal from the 
media processing system 125, in response to which the audio is played from an 
embedded speaker 530 in the printer 100. The electronic output system 120 may further 
include a player 525 or audio Tenderer that receives an encoded audio signal from the 
media processing system 125 and converts it into an audio signal for the speaker 530. 
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The player 525 thus takes some of the processing load off the media processing system 
125. For example, the player 525 may include a MIDI player for generating the audio 
signal; however, many other audio Tenderers may be used, in either hardware or 
software. 

[0045] In another embodiment, the electronic output system 120 includes a video 
display 535. The video display 535 is designed to receive a video signal from the media 
processing system 125, in response to which the video is played on the video display 535 
embedded in the printer 100. Similarly, the video display 535 may receive the video 
signal directly from a driver to reduce the processing load on the media processing 
system 125. 

[0046] In another embodiment, the printer 100 transmits the electronic output that is 
to be printed to another device as a signal. This signal can later be fixed in a tangible 

medium by the external device. To facilitate this, the electronic output system 120 

i 

includes a communication interface 540. The communication interface receives the 
electronic output from the media processing system 125 and sends the electronic output 
to the external device, which may be in communication with the printer 100 over a local 
network, the Internet, a wireless network, a direct connection, or any other suitable 
communication means. 

[0047] In another embodiment, the electronic output system 120 comprises an 
embedded web page display 545. The web page display 545 allows a user to see a 
representation of the electronic output in a web-based form. 
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Multimedia Processing System 

[0048] The multimedia processing system 125 of the printer 100 is designed to 
perform the specialized functionality of the multifunction printer 100. To send and 
receive messages between external devices or the user interface 110, the processing 
system 125 includes a processor 135 and a memory 130. In addition, the media 
processing system includes one or more hardware and/or software modules that enable 
the printer 100 to create related printed and electronic outputs for different types of 
multimedia data. In this way, the printer 100 can be configured to have a wide variety 
of multimedia processing functionalities. 

[0049] In one embodiment, as illustrated in FIG. 6, multimedia module 140 includes 
software and hardware to automatically detect 602 the coupling of a peripheral device 
150, a module to communicate with a peripheral device 604, a module to process and 
format multimedia data 606, and a module to generate an output 608. Multimedia 
module 140 is also configured to detect the presence of and communicate with a 
network device 170 in a similar manner. 

[0050] In an embodiment, as illustrated in FIG. 7, the device detection module 602 
automatically detects 702 the docking (or communicative coupling via a network) of a 
peripheral device 150 and downloads multimedia data from the peripheral device for 
processing by the printer 100. The multimedia processor 125 can utilize such protocols 
as the Plug and Play (PnP) or Universal Plug and Play (UPnP) protocol to automatically 
detect and communicate with devices that have PnP capabilities. With UPnP, a device 
can automatically communicate with other devices directly, convey its capabilities, and 
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learn about the presence and capabilities of other devices. Those skilled in the art will 
recognize that other detection methods, such as polling techniques, may be used to 
automatically detect the coupling of a peripheral device. Alternative systems for 
connection, such as IEEE 1394 cabling or Universal Serial Bus cabling, have equivalent 
standards for device and capability discovery. Alternatively, one embodiment detects 
the presence of an active electrical circuit in a physical connector, such as an RS232 serial 
port connector, between the docked device and printer 100. 

[0051] Once a device is docked to the printer 100 via peripheral interface 105 or 
network interface 157, the device communication module 604 is configured to 
communicate with the peripheral device 150. Again, PnP capabilities of a device will 
allow the communication module 604 to communicate with the device. The 
communication module 604 may be configured with specific software drivers to enable 
communication with specific peripheral devices. In one embodiment, the 
communication module 604 enables the printer 100 to operate a docked peripheral 
device to capture multimedia data. In such an embodiment, the printer 100 can also 
send commands to the docked peripheral device 150 and operate the device by 
controlling certain functionalities of the device. For example, the printer 100 could 
communicate with a docked cell phone to automatically bill the owner of the phone 
through her cell phone provider, or based on information in an XML profile on the 
phone. In another example, the printer 100 can communicate with a docked cell phone 
to issue commands to take pictures with the cell phone's digital camera or to call a 
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number from the cell phone. The picture might show who is standing at the printer 100. 
That data could be transmitted to some other destination by the cell phone when it calls 
out. The called number could be provided in a profile stored on the cell phone, or it 
could be stored on the printer. Other examples are discussed below. 
[0052] The communication module 604 sends 706 a request to the peripheral device 
for multimedia data to be downloaded to the printing system 100. Once the multimedia 
processing system 125 receives the multimedia data, the multimedia processing and 
formatting module 606 contains logic to format and process the multimedia data. In an 
embodiment, the format may be pre-defined and configured for specific peripheral 
devices. In other embodiments, a user may design a layout format for the multimedia 
data via the user interface 110. In yet another embodiment, a user may upload a 
predefined format. The multimedia processing and formatting module 606 provides an 
organized representation of the multimedia data depending on a particular peripheral 
device 150. For example, the formatter 606 may generate a format suitable for 
representation on printed output 160. Thus, if a video camcorder is docked directly to 
the printer, the format might comprise key frames with bar codes linking the key frames 
to different segments of a video file that may be used to replay a recorded video 
describing the event. Although the media processing system 125 is configured to 
perform at least some of the processing of the media data on the printer 100, it is 
preferably coupled to an external computing device that shares some of the computing 
burden. Network interface 157 allows communication with external network device 170 
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and/or external service 180, which are capable of performing at least a portion of the 
multimedia processing functionality. The network device 170 may be a computer 
system or a dedicated media processing hardware device. In this way, the printer 100 
relieves the source of the multimedia from at least some of the processing burden 
required to implement the printer's functionality, but the printer 100 need not shoulder 
the entire burden. The printer 100 can thus avoid slow-downs that can result from a 
heavy processing load, which may be especially important for shared printers. In an 
alternative embodiment, a peripheral device might initiate the download process, by 
requesting a connection with communication module 604. In this embodiment, the 
peripheral device might have a dedicated hardware button or a software interface which 
allows a user to initiate a transfer process by performing some action. 

Printer With Multimedia Functionality 

[0053] Printer 100 may include an embedded multimedia server module 610 that 
enables the printer 100 to act as a multimedia server and have associated functionality. 
In various embodiments, the multimedia server module 610 (610 is not shown in Fig. 6) 
includes hardware and software for carrying out multimedia functionality, media 
processing software, and computer interface hardware and software. In this way, the 
printer 100 can act like a multimedia server, which could be shared by a work group or 
used as a personal printer. Various embodiments of a multifunction printer having 
multimedia functionality are possible applications for a printer in accordance with 
embodiments of this invention. A number of specific embodiments for such a printer 
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are described in a co-pending U.S. patent application entitled "Printer with Multimedia 
Server," to Hull et. al, filed March 30, 2004, Attorney Docket 20412-8351, incorporated 
herein by reference in its entirety. 

[0054] In step 710, the multimedia output generator module 608 creates an output of 
the formatted multimedia data. The output may be sent to the printed output system 
115 where it will be printed on a document, or it may be sent to the electronic output 
system 120 where an electronic output will be produced. In one embodiment, the 
output may also be uploaded to a web server via network 155. 

Interactive Communication with a Printer 

[0055] FIG. 8 shows an example of interactive communication with a printer in 
accordance with an embodiment of the present invention. In general, conventional 
printer drivers in modern operating systems are not designed to facilitate interactive 
information gathering. Because a print job can be redirected to another printer, or the 
printing protocol does not allow such interactive sessions, the operating system does not 
encourage interaction with the user. Once initial printer settings are captured, further 
interactions are generally not allowed in conventional printers. One approach to this 
problem is to embed metadata into the print stream itself, as noted above. However, it is 
possible that the printer could need to ask the user for more information, in response to 
computations made from the data supplied by the user. In addition, the printer might 
itself delegate some tasks to other applications on other devices, which might in turn 
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need more information from the user. So-called ,r Web services" or "grid computing" 
systems are examples of the sort of application server that the printer might trigger. 
[0056] In order to allow this interaction, without modifying printer driver 
architecture of the underlying operating system, an extra mechanism, such as the one 
shown in Fig. 8, is constructed. A "UI Listener/' program 854 listens to a network 
socket, accepts requests for information 808, interacts with a user to obtain such data, 
and then sends the data back to the requester. 

[0057] Once a print request 802 is sent by user 850 and notification requested from 
the UI listener 804, the print job is sent by application 852. Here, the print job contains 
embedded information including the network address of the UI listener, authentication 
information, and the latest time that the client will be listening for requests. 
[0058] If the printer requires additional information of confirmation, it sends a 
request 808, which is detected by the UI listener,. which displays a dialog box to obtain 
input from the user 810. An example of such a request might be a request for a 
password or user confirmation code that the user must enter to access a database 858. 
The user" s input is included in a reply 812 sent to the printer. If the reply does not 
satisfy the printer it may ask for additional information (not shown). If the reply does 
satisfy the printer, it takes a next step. This step might be to perform an external action 
such as sending an email (not shown). The next step might also be sending a request for 
information 814 to an application server (such as a database) 858. In this example, 
application server 858 also sends a request for information 816, which is detected by the 
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UI listener 854. The user is prompted 818 and his response forwarded to the application 
server 820. In this example, a reply is then sent from the application server 858 to the 
printer 856. It will be understood that a particular embodiment may include either or 
none or requests 808 and 816 without departing from the spirit of the present invention. 
[0059] A program such as that shown in Fig. 8 may have a fixed set of possible 
interactions, or may accept a flexible command syntax that allows the requester to 
display many different requests. An example of such a command syntax is the web 
browser's ability to display HTML forms. These forms are generated by a remote server, 
and displayed by the browser, which then returns results to the server. In this 
embodiment, however, the UI listener is different from a browser in that a user does not 
generate the initial request to see a form. Instead, the remote machine generates this 
request. In the described embodiment, the UI listener is a server, not a client. 
[0060] Because network transactions of this type are prone to many complex error 
conditions, a system of timeouts preferably assures robust operation. Normally, each 
message sent across a network either expects a reply or is a one-way message. Messages 
which expect replies preferably have a timeout, a limited period of time during which it 
is acceptable for the reply to arrive. In this embodiment, embedded metadata includes 
metadata about a UI listener that will accept requests for further information. Such 
metadata preferably includes at least a network address, port number, and a timeout 
period. It can also include authentication information, designed to prevent malicious 
attempts to elicit information from the user. Because the user cannot tell whether the 
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request is coming from a printer, a delegated server, or a malicious agent, prudence 
suggests strong authentication by the UI listener. Information which specifies the correct 
authentication data is sent as a part of Request Notification 804, along with timeout 
information and the identity of the requesting printer, in one embodiment. If the printer 
or a delegated application server wishes more information, it can use the above noted 
information to request that the UI listener ask a user for the needed information. 
[0061] In one embodiment, printer 100 includes an interface 150 configured to 
support an image capture device. Examples of an image capture device include a digital 
still camera and a digital video camera. The user prints a document that contains 
instructions that drive the image capture device attached to the printer. The camera 
may be attached directly to the printer, e.g., via a cable, or may be wirelessly connected, 
for example using an 802.11 wireless networking connection. This allows the device to 
be located more remotely from the printer. The device can be instructed to take pictures 
or capture video immediately, or at some time in the future. The device is further 
instructed to print the captured pictures, either through the printed output system 115, 
the electronic output system 120, or both. 

[0062] In various embodiments, printing system 100 includes a variety of peripheral 
interfaces 105 and network interfaces 157. For example, in one embodiment, printing 
system 100 includes an interface for a wireless keyboard. In another embodiment, a 
interface is included for capturing audio. In a further embodiment, an RJ-11 pass- 
through is included in printing system 100, in order to monitor telephone activity and 
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print summary information about the calls. Attention is again drawn to U.S. patent 
application entitled, "Printer With Multimedia Server." 

[0063] Other embodiments provide interfaces with additional functionality. Some 
examples of additional functionality include a Bluetooth interface for recording and 
relaying data; a cellular telephone interface for downloading and printing images, audio 
and video clips from a cellular telephone; a hardware and software interface for a digital 
audio recorder or digital video recorder; a hardware and software interface for a 
portable or fixed-position meeting recorder, including those with pan, tilt and zooming 
capabilities; an interface for a head-mounted video camera with a gyroscope and GPS 
capture; an interface for capturing activity in an office environment; and an interface for 
capturing video from a frame buffer of a computer. 

[0064] In further embodiments, printing system 100 includes embedded technology 
such as a video ref ormatter; multimedia document formatter; video event detection 
logic; video foreground/background segmentation logic, face image detection, face 
image matching, facial recognition, and face extraction, matching and cataloging logic; 
video text localization and video OCR logic; video foreign language translation; video 
clip and video frame classification, including trainable video clip classification; digital 
image stitching; audio re-formatting; speech recognition; audio event detection; 
waveform matching; foreign language translation; caption alignment including caption 
alignment using video OCR; closed caption extraction and reformatting; speech 
recognition and closed caption plus transcript output; TV news segmentation and 
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formatting; music cataloging; video database logic; movie database logic; digital photo 
catalog logic; multimedia retrieval; audio and video clip segmentation and web 
publishing logic; world-wide web search logic; video clip retrieval and keyframe 
selection; image searching logic; weather map retrieval logic; aerial image retrieval logic, 
including image recognition and highlighting; streamlining media server logic; single 
and multi-channel streaming video monitoring and downloading; streaming audio 
monitoring, downloading and printing; meeting minutes extraction logic; video shot 
selection logic based on multiple audio sources; video clip segmentation logic including 
participant identification and cataloging; video clip segmentation based on audio event 
detection; embedded TV schedule traction and formatting logic; radio news 
segmentation and formatting logic; radio program segmentation logic including face 
image retrieval; radio program segmentation and automatic home page retrival; "books, 
on tape" speech recognition and formatting software, including foreign language 
translation and formatting; map generation software for routing; route planning logic; 
forms recognition logic; text-to-speech logic; document summarization and text-to- 
speech logic; graph printing logic; and RFID signaling apparatus and variable content 
selection logic. 

[0065] Further descriptions of the various embodiments described above are 
included in the following co-pending U.S. patent applications: "Printer With Hardware 
and Software Interfaces for Peripheral Devices," filed March 30, 2004, Attorney Docket 
No. 20412-08382; "Printing System With Embedded Audio/Video Content Recognition 
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and Processing," filed March 30, 2004, Attorney Docket No. 20412-08394; "Printer With 
Embedded Retrieval and Publishing Interface," filed March 30, 2004, Attorney Docket 
No. 20412-08421; "Printer with Radio or Television Program Extraction and Formatting," 
filed March 30, 2004, Attorney Docket No. 20412-08440; and "Printer With Document- 
Triggered Processing," filed March 30, 2004, Attorney Docket No. 20412-08449. Each of 
these references is incorporated herein in its entirety. 

[0066] While examples of suitable printing systems are described above, the 
description of the printer and its document production means is not meant to be 
limiting. Depending on the intended application, a printer can take many different 
forms other than the typical office or home-use printer with which most people are 
familiar. Therefore, it should be understood that the definition of a printer includes any 
device that is capable of producing an image, words, or any other markings on a surface. 
Although printing on paper is discussed above, it should be understood that a printer in 
accordance with various embodiments of the present invention could produce an image, 
words, or other markings onto a variety of tangible media, such as transparency sheets 
for overhead projectors, film, slides, canvass, glass, stickers, or any other medium that 
accepts such markings. 

[0067] In addition, the description and use of multimedia and multimedia data are 
not meant to be limiting, as multimedia may include any information used to represent 
any kind of media or multimedia content, such as all or part of an audio and/or video 
file, a data stream having multimedia content, or a transmission of multimedia content. 

31 



Multimedia content may include one or a combination of audio (including music, radio 
broadcasts, recordings, advertisements, etc.), video (including movies, video clips, 
television broadcasts, advertisements, etc.), software (including video games, 
multimedia programs, graphics software, etc.), and pictures; however, this listing is not 
exhaustive. Furthermore, multimedia data may further include anything that itself 
comprises multimedia content or multimedia data, in whole or in part, and multimedia 
data includes data that describes a real- world event. Multimedia data can be encoded 
using any encoding technology, such as MPEG in the case of video and MP3 in the case 
of audio. They may also be encrypted to protect their content using an encryption 
algorithm, such as DES, triple DES, or any other suitable encryption technique. 
[0068] Moreover, any of the steps, operations, or processes described herein can be 
performed or implemented with one or more software modules or hardware modules, 
alone or in combination with other devices. It should further be understood that 
portions of the printer described in terms of hardware elements may be implemented 
with software, and that software elements may be implemented with hardware, such as 
hard-coded into a dedicated circuit. In one embodiment, a software module is 
implemented with a computer program product comprising a computer-readable 
medium containing computer program code, which can be executed by a computer 
processor for performing the steps, operations, or processes described herein. 
[0069] In alternative embodiments, the printer can use multiple application servers, 
acting in cooperation. Any of the requests or messages sent or received by the printer 
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can be sent across a network, using local cables such as IEEE1394, Universal Serial Bus, 
using wireless networks such as IEEE 802.11 or IEEE 802.15 networks, or in any 
combination of the above. 

[0070] The foregoing description of the embodiments of the invention has been 
presented for the purpose of illustration; it is not intended to be exhaustive or to limit 
the invention to the precise forms disclosed. Persons skilled in the relevant art can 
appreciate that many modifications and variations are possible in light of the above 
teachings. It is therefore intended that the scope of the invention be limited not by this 
detailed description, but rather by the claims appended hereto. 
[0071] We claim: 
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